2. Kullback-Leibler divergence

The KL divergence evaluates the difference of two different distributions, It's often a powerful tool for make two distributions coincide in the training of deep-learning neural networks.

Reading time

3 min

Length

465 words

Updated

Sep 15, 2026

Total views

--

#math/KL-divergence

1. Definition

KL divergence is an important way to evaluate the similarity of 2 distributions, then give a reasonable Loss. Thus it is often used as the learning function of generated network, for example, VAE (Variational autoencoder)

(1) Concepts of KL-divergence

The Kullback-Leibler divergence(KL divergence)^[1] is also called relative entropy or I-divergence, is a type of statistical or distribution distance. which can measure one distribution $Q$ is different from true probability $P$.

$$\begin{array}{}\text{(1.1)}& \boxed{{\displaystyle D_{KL}(P||Q)=\sum _{x\in \mathcal{X}}P(x){\log }_{}\frac{P(x)}{Q(x)}}}\end{array}$$

or equivalent to :

$$\begin{array}{}\text{(1.2)}& \boxed{{\displaystyle D_{KL}(P||Q)=-\sum _{x\in \mathcal{X}}P(x){\log }_{}\frac{Q(x)}{P(x)}}}\end{array}$$

Also it is  expectation of the logarithmic difference between the probabilities $P$ and $Q$ using the probabilities $P$.

For continuous function, it is also called relative entropy (defined as integral):

$$\begin{array}{}\text{(1.3)}& D_{KL}(P||Q)={\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}p(x){\log }_{}\frac{P(x)}{Q(x)}=E_{x\sim p}[{\log }_{}\frac{P(x)}{Q(x)}]\end{array}$$

Note $D_{KL}$ always $\ge 0$.

proof of positiveness of KL-divergence Using Jensen's inequality, we have, for a strict convex function :

$$E[f(x)]\ge f(E[x])$$

Then we have :

$$D_{KL}\ge {\log }_{}\left[E_p\left(\frac{P(x)}{Q(x)}\right)\right]=-{\log }_{}\int q(x)dx=-{\log }_{}1=0$$

(2) Design Thoughts of KL-divergence

In the application purpose, we want the $q$ to approximate the distribution of $p$, So, we care about that : The q should be high where p is high :

Since we care more about the q is similar to p

Intuitively, there are three cases of importance^[2] (changed some expressions for consistency): • If $p$ is high and $q$ is high, then we are happy (i.e. low KL divergence). • If $p$ is high and $q$ is low then we pay a price (i.e. high KL divergence). • If $p$ is low, then we don't care (i.e. also low KL divergence, regardless of q).

(3) Code Implementation

The most common case is to compute the KL_divergence between two normal distributions. In that case, from (1.1)^[3], we know :

$$\begin{array}{rl}{D}_{KL}(P||Q)=& {\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}\frac{1}{{\sigma }_1\sqrt{2\pi }}\exp (-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2})[{\log }_{}\left(\frac{{\sigma }_2}{{\sigma }_1}\right)-\frac{(x-{\mu }_1{)}^2}{2{\sigma }_1^2}+\frac{(x-{\mu }_2{)}^2}{2{\sigma }_2^2}]dx\\ =& {\log }_{}\left(\frac{{\sigma }_2}{{\sigma }_1}\right)-\frac{1}{2{\sigma }_1^2}\mathbb{E}[(x-{\mu }_1{)}^2]+\frac{1}{2{\sigma }_2^2}\mathbb{E}[(x-{\mu }_2{)}^2]\end{array}$$

For the first term, we have that :

$${\mathbb{E}}_{X\sim N}[(X-{\mu }_1{)}^2]=\int (X-{\mu }_1{)}^{2}p(X)dx={\sigma }_1^2\to -\frac{1}{2{\sigma }_1^2}\mathbb{E}[(x-{\mu }_1{)}^2]=-\frac{1}{2}$$

Then for the second term :

$$\mathbb{E}[(x-{\mu }_2{)}^2]=\mathbb{E}[(x-{\mu }_1{)}^2+\cancel{2(x-{\mu }_1)({\mu }_1-{\mu }_2)}+({\mu }_1-{\mu }_2{)}^2]={\sigma }_1^2+({\mu }_1-{\mu }_2{)}^2$$

So the final term reduce to :

$$D_{KL}(P||Q)={\log }_{}\left(\frac{{\sigma }_2}{{\sigma }_1}\right)+\frac{1}{2{\sigma }_2^2}({\sigma }_1^2-{\sigma }_2^2+({\mu }_1-{\mu }_2{)}^2)$$

def kl_divergence(
    mu1: torch.Tensor, logvar1: torch.Tensor,
    mu2: torch.Tensor, logvar2: torch.Tensor
):
    """
    :param mu1:
    :param logvar1:
    :param mu2:
    :param logvar2:
    :return:
    """
    fst_term = 1/2 * (logvar1 - logvar2)
    var1 = torch.exp(logvar1)
    var2 = torch.exp(logvar2)
    scd_term = 1/(2 * var2) * (var1 - var2 + (mu1 - mu2) ** 2)
    return fst_term + scd_term

We note another version can be found as normal_kl function in topodiff ^[4], which has similar implementation, and reach the exact same results.

def normal_kl(mean1, logvar1, mean2, logvar2):
    """
    Compute the KL divergence between two gaussians.

    Shapes are automatically broadcasted, so batches can be compared to
    scalars, among other use cases.
    """
    tensor = None
    for obj in (mean1, logvar1, mean2, logvar2):
        if isinstance(obj, th.Tensor):
            tensor = obj
            break
    assert tensor is not None, "at least one argument must be a Tensor"

    # Force variances to be Tensors. Broadcasting helps convert scalars to
    # Tensors, but it does not work for th.exp().
    logvar1, logvar2 = [
        x if isinstance(x, th.Tensor) else th.tensor(x).to(tensor)
        for x in (logvar1, logvar2)
    ]

    return 0.5 * (
        -1.0
        + logvar2
        - logvar1
        + th.exp(logvar1 - logvar2)
        + ((mean1 - mean2) ** 2) * th.exp(-logvar2)
    )

---

1. https://en.wikipedia.org/wiki/Kullback–Leibler_divergence ↩︎

2. https://www.cs.cmu.edu/~epxing/Class/10708-17/notes-17/10708-scribe-lecture13.pdf ↩︎

3. 1. Summary for basics of probability theory-Prior and Posterior Probability ↩︎

4. https://github.com/francoismaze/topodiff/blob/main/topodiff/losses.py ↩︎